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THE  PROBLEM 


Roughly  5%  of  student  naval  aviators  fail  the  advanced  phase  of  flight  training.  At 
this  stage  of  training,  the  Navy  has  spent  between  $300,000  and  $1,000,000  per  student. 
Any  reduction  in  this  attrition  rate  through  prior  screening  would  be  of  great  economic 
benefit  to  the  Navy.  Computer-based  performance  tests  developed  at  the  Naval  Aerospace 
Medical  Research  Laboratory  (NAMRL)  were  assessed  to  determine  whether  they  could 
augment  the  present  medical  screening  standards  and  thereby  help  identify  potential 
failures  in  advanced  flight  training. 

FINDINGS 


A  weak  statistical  relationship  exists  between  a  dual-task  performance  test,  accession 
source,  college  major,  an  aptitude  test,  and  success  in  advanced  flight  training.  Discrim¬ 
inant  analysis  was  employed  to  find  a  linear  composite  score  of  these  variables  that  could 
be  used  to  classify  a  student  as  a  probable  pass  or  fail  in  advaiiced  flight  training.  For 
example,  the  model  presented  m  this  report  could  reduce  failures  by  50%  at  the  cost  of 
rejecting  roughly  20%  of  those  students  who  eventually  passed.  A  Bayesian  analysis  of 
the  success  rate  parameter  showed  that  this  particular  model  did  result  in  a  significant 
improvement  over  the  present  selection  system. 


RECOMMENDATIONS 


These  data  can  be  used  to  make  cost-benefit  tradeoffs  for  aviation  selection  policy 
making.  The  author  recommends  that  the  dual-task  performance  test  and  accompanying 
statistical  model  discussed  in  this  report  be  considered  for  operational  implementation 
as  part  of  an  improved  medical  selection  process  for  potential  Navy  and  Marine  Corps 
aviators. 
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INTRODUCTION 


In  a  previous  report  [1]  computer-based  performance  tests  were  evaluated  on  how  well 
they  could  predict  the  success  of  student  naval  aviators  in  the  primary  phase  of  flight 
«  training.  Thereafter,  the  progress  of  these  student  aviators  was  monitored  throughout  the 

flight  training  program  through  the  receipt  of  criterion  data  on  grades,  success,  pipeline, 
et  cetera^  from  Chief,  Naval  Aviation  IVaining  (CNATRA)  in  Corpus  Christi,  Texas. 

When  the  data  received  during  the  intermediate  and  advanced  phases  of  flight  training 
were  subsequently  analyzed,  no  significant  results  were  found  between  our  battery  of 
^  tests  and  success  in  the  intermediate  phase  of  flight  training.  We  did,  however,  uncover 

a  significant  association  between  a  dual-task  performance  test  and  success  in  advanced 
flight  training. 

These  results  axe  presented  using  a  different  method,  as  compared  to  the  first  report, 
for  quantifying  the  importance  of  the  data.  Specifically,  a  linear  composite  of  the  test 
battery  scores  is  constructed  by  using  discriminant  analysis.  The  discriminant  function 
scores  are  then  used  to  classify  the  students  as  to  success  or  failure  in  advanced  flight 
training.  The  success  rate  using  these  tests  is  then  compared  to  the  success  rate  under  the 
present  system.  The  formal  technique  employed  here  for  making  this  comparison  is  the 
traditional  Bayesian  approach  of  calculating  posterior  probability  density  functions  for 
the  success  rate  parameter.  Finally,  confidence  intervals  are  used  to  show  the  quantifiable 
consequences  for  Navy  planners  if  they  should  decide  to  use  these  tests  for  selection 
purposes. 

A  coxmter-argument  based  on  the  hypothetical  inflexibility  of  the  training  command 
infrastructure  due  to  fixed  costs  is  presented  in  the  Discussion.  If  such  an  argument  has 
any  merit,  it  tends  to  vitiate  any  optimism  for  implementing  these  tests  for  reasons  due 
solely  to  the  statistical  results. 


METHODS 


SUBJECTS 

Student  naval  aviators,  preselected  for  naval  aviation  flight  training  on  the  basis  of 
their  performance  on  the  current  Navy  and  Marine  Corps  aviation  selection  tests  and 
medical  examinations,  participated  in  the  study.  The  actual  number  of  subjects  for 
specific  cases  axe  presented  in  the  Results  section.  The  subjects  were  informed  that  a) 
the  investigation  involved  performing  tasks  in  problem  solving  and  perceptual  and  motor 
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skills,  b)  their  test  performance  would  not  aifect  their  continuation  in  the  program  nor 
be  entered  into  their  permanent  service  records  and,  c)  results  would  be  used  solely  for 
the  purpose  of  developing  an  improved  aviation  selection  program  for  the  Navy. 

APPARATUS 

All  testing  was  conducted  on  Apple  He  microcomputers  with  Apple  monochrome  moni¬ 
tors  (CRTs).  Subjects  used  a  numeric  keypad  to  respond  to  discrete  stimuli.  All  responses 
were  recorded  to  millisecond  accuracy.  A  Measurement  Systems  Incorporated  (MST542) 
control  stick  was  used  for  joystick  and  throttle  control  during  the  tracking  tasks.  Rud- 
derpedal  controls  were  measured  using  a  variable  resistor  connected  to  a  computer  A/D 
channel.  The  joystick  was  mounted  on  the  forward  edge  of  the  testing  console  at  a 
centered  position.  The  throttle  was  located  on  the  left  side  of  the  testing  booth.  The 
rudderpedals  were  located  so  that  the  subjects  could  easily  place  both  feet  while  sitting 
in  the  testing  booth.  Subjects  operated  the  joystick  with  the  right  hand,  the  throttle 
with  the  left  hand,  and  the  rudderpedals  with  both  feet. 

PROCEDURE 

All  candidates  were  tested  before  entering  flight  training  and  after  completing  a  14- 
week  basic  military  indoctrination  program  for  AOC  officers  or  a  6-week  program  for  stu¬ 
dents  already  commissioned.  All  instructions  were  presented  to  the  subjects  on  the  CRT 
for  each  task  individually.  Test  administrators  intervened  only  to  begin  the  computer 
program  for  each  task  and  to  answer  questions  posed  by  subjects.  The  test  administra¬ 
tion  time  of  the  battery  ranged  from  3.7  to  4.0  h.  The  order  of  the  tasks  and  the  stimuli 
within  each  test  were  identical  for  all  subjects.  Subjects  received  a  3-4  min  rest  period 
betweeri  tasks.  All  testing  took  place  in  an  air-conditioned  laboratory. 


PREDICTOR  TESTS 


The  entire  complement  of  tests  in  the  NAMRL  computer-based  performance  test  bat¬ 
tery  is  described  in  [1].  We  repeat  the  description  of  the  Horizontal  TVacking  test,  the 
Absolute  Difference  test,  and  the  combined  Absolute  Difference- Horizontal  Tracking  test 
because  these  tests  exhibited  an  association  with  success  in  advanced  flight  training. 
Damos  and  Gibb  [2]  explain  the  rationale  for  these  tests  and  present  some  preliminary 
results  comparing  the  performance  of  experienced  fleet  pilots  to  student  naval  aviators 
on  these  tests. 

Several  background  variables  were  recorded  for  each  subject.  These  included: 
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1.  Initial  aviation  selection  test  scores,  t.e.  Academic  Qualification  Test  (AQT) 
and  Flight  Aptitude  Rating  (FAR) 

2.  Previous  civilian  flight  training 

3.  Age 

4.  Source  of  procurement 

5.  Gender 

6.  College  major 

Previous  civilian  flight  training  was  treated  as  a  continuous  variable  using  the  self- 
reported  number  of  total  flight  hours  a  subject  had  logged.  Total  flight  hours  included 
both  solo  and  dual  pilot  training  hours. 

Source  of  procurement  included  six  distinct  groups: 

1.  Aviation  Officer  Candidate  School  (college  graduates  entering  directly  into 
the  military) 

2.  Naval  Academy  graduates 

3.  Naval  cadets  (prior  enlisted  service  or  60  college  credits  with  numerous  other 
criteria) 

4.  Marine  Corps  Officer  Program 

5.  Naval  Reserve  Officer  Training  Corps 

6.  Other  (direct  procurement,  Merchant  Marine  Academy,  enlisted  commis- 
•  sioning  programs) 

College  major  was  classified  into  one  of  five  general  disciplines: 

1.  Engineering  and  math 

2.  Physical  sciences  (biology,  geology,  physics,  etc.) 

3.  Business 

4.  Social  sciences  (psychology,  sociology,  history,  etc.) 

5.  Physical  education 
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HORIZONTAL  TRACKING 


In  this  test,  the  subject  was  required  to  learn  a  one-dimensional  tracking  task.  To 
perform  the  test  successfully,  the  subject  had  to  anticipate  the  movement  of  a  square 
on  the  computer  screen  and  manipulate  the  joystick  to  counterbalance  the  movement  in 
order  to  keep  the  square  centered  on  a  fixed  central  point  on  the  screen.  For  example,  if 
the  square  was  moving  off  center  to  the  right,  the  subject  would  move  the  joystick  to  the 
left  in  order  to  re-center  the  square.  Specifically,  the  subject  maintained  a  0.6-cni  square 
centered  in  a  9.75  by  1.25-cm  rectangle  by  moving  the  joystick  either  left  or  right.  The 
square  was  driven  by  a  forcing  function  programmed  into  the  computer.  Each  subject 
received  ten  2-min  trials  separated  by  a  30-s  rest.  The  dependent  measure  was  RMS  error. 
Total  testing  time  was  25-min.  This  test  was  designed  to  measure  compensatory  tracking 
skills.  A  compensatory  tracking  task  is  defined  as  one  in  which  only  the  difference  (the 
error)  between  the  command  input  and  the  system  output  is  displayed.  [3] 

ABSOLUTE  DIFFERENCE  TASK 

In  this  test,  the  subject  was  visually  presented  with  a  random  number  between  1 
and  9  on  the  CRT.  This  number  disappeared  from  the  screen  and  was  then  followed 
immediately  with  another  number.  The  subject  was  required  to  press  the  correct  button 
on  the  numeric  keypad  to  indicate  the  absolute  difference  between  the  number  presently 
displayed  on  the  CRT  screen  and  the  number  shown  on  the  screen  in  the  previous  trial. 
The  numbers  were  presented  in  such  a  manner  that  only  absolute  differences  from  1  to  4 
were  correct.  When  the  subject  made  a  response,  a  new  number  appeared  on  the  CRT, 
«md  again  the  subject  calculated  the  absolute  difference  between  the  nmnber  previously 
presented  and  the  number  currently  displayed.  The  subject  was  instructed  to  use  only 
his  right  hand  in  responding  with  the  numeric  keypad.  Speed  imd  accuracy  of  response 
were  equally  eInph^lsized.  This  test  was  self-paced  and  consisted  of  fifteen  2-min  trials. 
Total  testing  time  was  30  min.  The  dependent  measures  consisted  of  number  correct, 
number  of  errors,  RT  on  correct  responses,  and  RT  on  error  responses.  This  task  was 
essentially  a  measure  of  short-term  memory,  memory  search,  and  encoding. 

ABSOLUTE  DIFFERENCE  AND  HORIZONTAL  TRACKING 

As  part  of  the  emphasis  on  time-sharing  ability  within  the  NAMRL  battery,  the  hor¬ 
izontal  tracking  and  absolute  difference  tasks  were  combined  for  a  measure  of  dual-task 
performance.  For  this  combination  of  tasks,  the  subject  performed  the  horizontal  track¬ 
ing  task  and  the  absolute  difference  task  simultaneously.  The  ntnnber  for  the  absolute 
difference  tsisk  was  centered  above  the  tracking  task  and  touched  the  top  of  the  tracking 
task.  The  subject  controlled  the  tracking  task  with  his  right  hand  and,  for  the  absolute 
difference  task,  pressed  a  number  on  the  keypad  with  his  left  hand.  The  subject  was 
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instructed  that  the  two  tasks  were  equally  important.  The  subject  received  five  2-min 
trials  separated  by  rest  periods  of  30-60  s.  Total  testing  time  for  the  dual-task  combina¬ 
tion  was  15  min.  Figure  1  depicts  this  dual-task  test  as  it  appeared  to  the  subject  on  the 
computer  screen. 


Figure  1.  The  one-dimensional  tracking  test  and  absolute  difference  test  combined  tc  form  a  dual-task 
test. 


RESULTS 


DATA  BASE  AND  PREPROCESSING  OF  VARIABLES 

The  original  data  base  consisted  of  144  variables  on  1293  student  naval  aviators 
(SNAs).  Since  the  previously  cited  report  on  primary  flight  training,  additional  data 
from  CNATRA  have  been  received  on  the  intermediate  and  advanced  phases  of  flight 
training.  When  these  data  were  added  to  the  data  base,  there  was  pass  or  fail  infor¬ 
mation  for  the  advanced  phase  of  flight  training  on  836  SNAs.  There  was  a  substantial 
rimount  of  missing  data  for  these  836  subjects  because  individual  tests  were  introduced 
into  the  test  battery  at  different  points  in  time. 

Appendix  A  contains  descriptive  data  in  the  form  of  cross-tabulation  tables  for  success 
in  advanced  flight  training  versus  selected  background  variables.  The  frequency  table  for 
Training  Success  is  presented  first,  followed  by  the  cross-tabulation  tables  for  Training 
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Success  versus  Gender,  Accession  Source,  College  Major,  AQT,  FAR,  and  Age.  The 
frequencies  for  these  variables,  when  the  missing  data  are  included,  add  up  to  836. 

In  order  to  cope  with  this  mass  of  data,  we  used  hierai’chical  multiple  regression 
to  screen  out  promising  variables  for  attention.^  When  the  dichotomous  variable  of 
success  in  advanced  flight  training  was  employed  as  the  criterion  variable,  only  one  set 
of  variables  reached  statistical  significance.  After  the  variables  of  age,  previous  flight 
hours,  sex,  accession  group,  college  major,  AQT,  and  FAR  had  been  entered  in  the 
regression  equation,  only  the  eight  variables  from  the  AD,  HT,  and  dual-task  ADHT 
reached  significance  as  measured  by  the  increase  in  B?.  [4] 

At  this  point,  the  number  of  variables  had  been  pared  to  15.  We  subjected  these  15 
variables  to  fimther  analysis.  First,  we  desired  to  construct  a  composite  score  based  on 
the  eight  ADHT  variables.  One  feasible  composite  score  was  arrived  at  by  utilizing  only 
the  dual-task  ADHT  test  in  the  following  manner: 

ADHTCS  =  .20  ♦  ZADHT6  -  .50  *  ZADHT5  -  .10  *  ZADHT7  -  .20  ♦  ZADHT8 

This  composite  score  consisted  only  of  the  four  variables  from  the  dual-task  ADHT  test, 
i.e.  when  subjects  were  performing  the  absolute  difference  task  and  the  horizontal  track  ¬ 
ing  task  at  the  same  time.  Because  the  subjects  had  been  instructed  to  pay  equal  atten¬ 
tion  to  both  parts  of  the  test,  the  composite  score  weighted  the  one  variable  concerned 
with  tracking  error  (ZADHT5)  equally  with  the  three  variables  concerned  with  absolute 
difference  performance  (ZADHT6,  ZADKT7,  and  ZADHT8).  These  last  three  variables 
together  were  given  a  weight  equal  to  0.5  with  the  further  breakdown  that  mean  nmnber 
of  errors  (ZADHT7)  was  given  only  half  the  weight  of  meaii  number  correct  (ZADHT6) 
and  mean  correct  RT  (ZADHT8).  All  four  scores  were  first  converted  to  z-scores  be¬ 
fore  the  composite  score  was  formed.  Any  constituent  z-score  making  up  the  ADHTCS 
greater  than  4  in  absolute  value  was  rejected  as  an  outlier. 

Success  in  advanced  flight  training  was  coded  as  a  “1”  and  failure  as  a  “0.”  Perfor¬ 
mance  on  AD  number  correct  (7ADHT6)  was  the  only  variable  positively  correlated  with 
the  success  code.  The  other  tliree  variables,  tracking  error  (ZADHT5),  AD  number  of 
enors  (ZADHT7),  eind  mean  correct  RT  (ZADHT8),  were  negatively  correlated  with  the 
success  code.  This  pattern  of  correlations  explains  the  signs  of  the  weighting  coefficients 
in  the  composite  score  so  that  ADHTCS  as  constructed  was  positively  correlated  with 
the  success  code. 

To  illustrate,  consider  the  computation  of  the  ADHT  composite  score  for  a  hypothet¬ 
ical  SNA  who  did  much  better  than  average  on  the  tracking  portion  of  the  test,  but 
performed  somewhat  poorer  on  the  absolute  difference  segment.  His  z-score  for  tracking 
(ZADHT5)  was  —2.  (Remember  that  a  lower  z-score  for  tracking  error  represents  better 
performance).  On  the  other  hand,  his  z-score  for  mean  nmnber  correct  on  the  absolute 

’This  and  other  related  work  was  carried  out  by  Dr.  Harold  Delaney  of  the  University  of  New  Mexico  while  be  held  an 
ASEB  Sununer  Faculty  appointment  at  NAMRL  in  1991. 
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difference  segment  (ZADHT6)  was  —1,  his  z-score  for  mean  number  of  errors  (ZADHT7) 
was  +1.3  (a  higher  z-score  represents  poorer  performance),  and  his  mean  correct  RT 
(ZADHT8)  was  +1  (again,  a  higher  z-score  represents  poorer  performance).  When  these 
■'.^ues  are  inserted  into  the  composite  score  equation,  ADHTCS  equals  +0.47. 

Furthermore,  of  the  background  variables  and  aptitude  tests,  only  accession  group, 
college  major,  and  AQT  played  a  significant  role.  The  results  reported  here  concern 
these  three  variables  and  the  ADHT  composite  score.  Since  accession  group  and  college 
major  were  both  coded  with  two  dummy  variables,  the  analysis  will  deal  with  six  variables 
in  all. 

DISCRIMINANT  ANALYSIS 

A  discriminant  analysis  (DA)  was  conducted  on  these  variables  with  the  SPSS/PC+ 
statistical  package  [5].  Only  cases  with  complete  data  on  all  variables  were  subjected  to 
analysis.  Of  451  such  cases,  432  students  passed  advanced  flight  training  (PASS),  and  19 
students  failed  advanced  training  (FAIL). 

There  were  438  males  and  13  females  constituting  the  451  SNAs  with  complete  data. 
Their  age  ranged  from  20  to  29  with  a  mean  age  of  23.04  and  a  standard  deviation  of 
1.43. 

The  relevant  summary  statistics  from  the  DA  were  as  follows.  The  canonical  cor¬ 
relation  equaled  .1712,  indicating  a  slight  influence  of  these  six  variables  on  success  in 
advanced  flight  training.  A  =  13.274  suggests  that  the  means  of  the  discriminant 
function  scores  for  the  PASS  group  and  the  PAIL  group  were  significantly  different  from 
zero  (p  <  .039,  6df).  The  covariance  matrices  for  the  two  groups  did  not  depart  from 
equality  by  Box’s  M  test. 

Table  1  illustrates  in  detail  the  calculation  of  the  discriminant  score  for  one  SNA. 
The  six  variables  plus  a  constant  are  shown  in  the  first  column.  di  and  d2  are  dummy 
variables  coding  accession  group,  while  da  and  are  dummy  variables  encoding  college 
major.  Table  2  shows  the  relationship  between  these  variables  and  their  dummy  variable 
coding.  The  second  column  of  Table  1  gives  the  unstandardized  discrimination  function 
coefficients  as  computed  by  the  program.  The  third  column  shows  the  values  for  a 
hypothetical  subject  on  the  six  variables.  This  subject  had  an  AQT  stanine  score  of 
7,  (on  a  scale  of  1-9),  and  was  a  Naval  Academy  accession  (dj  =  0  dj  =  0)  with  an 
engineering/mathematics  degree  (da  =  1  d4  =  0).  We  shall  use  the  previously  computed 
ADHT  composite  score  of  .47  for  this  SNA. 

The  mean  discriminant  score  for  the  entire  FAIL  group  in  the  sample  was  .827,  and 
the  mean  discriminant  score  for  the  entire  PASS  group  was  —.036.  The  mathematical 
details  of  how  the  students  axe  classified  into  a  predicted  PASS  or  FAIL  group  based 
on  the  discriminant  score  are  given  in  appendix  B.  For  the  hypothetical  SNA  given  in 
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Table  1.  Unstandardized  discrimination  function  coefficients  and  values  for  six  variables  plus  a  constant 
needed  to  compute  a  discriminant  function  score  for  a  hypothetical  subject. 


Variable 

Coefficient 

Value 

Mul tiplication  (xi) 

AQT 

.263 

7 

1.843 

di 

.586 

0 

0.000 

C?2 

-.238 

0 

0.000 

ds 

1.474 

1 

1.474 

di 

.735 

0 

0.000 

ADHTCS 

-1.324 

0.47 

-0.622 

Constant 

-2.531 

1 

-2.531 

Discriminant  Function  Score  x,  =  .164 


Table  2.  The  dummy  coding  for  the  accession  source  and  college  major  variables. 


Accession  Somce 

□ 

Jz 

Naval  Academy 

0 

0 

AOC/OCS 

1 

0 

ROTC/Maxines 

0 

1 

College  Major 

ds 

General  Science 

0 

0 

Engineering/Math 

1 

0 

Liberal  Arts 
_ 1 

0 

1 

8 


Table  3,  The  classificatiou  mat.  'x  arising  from  the  discriminant  analysis  of  451  SNAs  showing  the  predicted 
passes  and  fails  of  the  model  compared  to  the  actual  data. 


PREDICTED 


Pass 

Fail 

Pass 

339 

93 

ACTUAL 

Fail 

9 

10 

348 

103 

432 

19 

451 


Table  1,  the  calculations  in  appendix  B  assign  a  probability  of  passing  advanced  flight 
training  of  63%.  The  SNA  would,  therefore,  be  assigned  to  the  PASS  group. 

Table  3  shows  the  classification  matrix  that  is  produced  when  the  DA  program  classifies 
eadi  of  the  451  known  pass  or  fails  into  a  predicted  pass  or  fail.  The  prior  probabilities 
parameter  in  the  DA  program  was  adjusted  so  that  close  to  a  50%  attrition  rate  would 
be  achieved.  Prom  Table  3,  ob.serve  that  10  of  19  students  have  been  correctly  predicted 
into  the  fail  category.  The  observed  success  rate  of  97.41%  (339/348),  when  using  these 
variables  to  discriminate  the  PASS  and  FAIL  groups,  is  achieved  at  the  cost  of  rejecting 
93  SNAs  who  otherwise  would  have  passed. 

Figure  2  shows  the  theoretical  Gaussian  distribution  of  the  discriminant  function 
scores.  The  means  axe  set  to  the  values  as  calculated  by  the  program,  Up  ass  =  —.03637 
and  Ufail  —  .82694,  with  apAss  =  <^fail  =  !•  The  threshold  discriminant  score  at  .769 
divides  the  two  curves  into  four  areas.  The  nmnber  of  SNAs  falling  into  each  of  these 
four  areeis  when  the  threshold  discrinxinant  function  score  is  used  to  classify  SNAs  is  also 
shown.  This  is  the  same  information  presented  in  the  classification  matrix  of  Table  3. 

Logistic  regression  is  sometimes  recommended  as  an  alternative  to  discriminant  anal¬ 
ysis  for  finding  the  weights  to  attach  to  the  variables  that  form  the  linear  combination 
used  subsequently  to  classify  cases.  The  logistic  model  may  be  more  appropriate  when 
the  explanatory  variables,  such  as  accession  source  and  college  major,  are  qualitative  and 
obviously  not  distributed  as  midtivariate  normal. 

Accordingly,  a  logistic  regression  weis  carried  out  using  the  same  six  explanatory  vari¬ 
ables  as  in  the  DA  model  and  with  training  success  as  the  dependent  variable.  The 
logistic  regression  parameters  were  estimated  by  maximrim  likelihood.  These  parame- 
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threshold =.769 


Discriminant  Function  Scores 


Figure  2.  The  theoretical  Gaussian  probability  density  functions  of  the  discriminant  function  scores  for 
the  PASS  and  FAIL  groups  in  advanced  flight  training. 
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Table  4.  Logistic  regression  coefficients  for  the  same  six  variables  and  hypothetical  values  used  in  the  DA 
model.  The  logistic  regression  coefficients  are  meant  to  be  compared  with  the  DA  coefficients  shown  in 
Table  1. 


Variable 

Coefficient 

Value 

Multiplication 

AQT 

-.231 

7 

-1.614 

di 

-.566 

0 

0.000 

dj 

.246 

0 

0.000 

d. 

-1.690 

1 

-1.690 

-1.068 

0 

0.000 

ADHTCS 

1.071 

0.47 

.503 

Constant 

6.065 

1 

6.065 

Logistic  regression  score 

ELi^.  =  3.26 

ter  estimates  are  presented  in  Table  4  where  they  may  be  compared  to  those  found  in 
Table  1.  Except  for  a  change  in  sign  and  a  shift  due  to  the  constant,  these  coefficients 
are  comparable  to  the  coefficients  found  using  the  DA  estimates.  The  same  hypothetical 
SNA  is  used  to  compute  a  logistic  regression  score  of  3.26  when  using  these  coefficients. 

The  mean  of  the  logistic  regression  scores  was  3.522  for  the  PASS  group  and  2.745 
for  the  FAIL  group.  The  posterior  probability  of  belonging  to  the  PASS  group  for  a 
given  logistic  score  was  calculated,  as  explained  in  appendix  B,  in  the  same  fashion  as 
for  the  DA  model.  The  hypothetical  SNA  with  the  logistic  regression  score  of  3.26  has 
a  posterior  probabihty  of  60%  of  passing  as  compared  to  63%  for  the  DA  model.  This 
SNA  would  still  be  classified  as  belonging  to  the  PASS  group. 

The  classification  matrix  shown  in  Table  5  results  when  these  probabilities  are  used  to 
classify  all  451  SNAs.  We  can  see  that  the  classification  based  on  the  use  of  the  logistic 
regression  weights  hardly  differs  from  the  outcome  based  on  the  DA  weights. 


BAYESIAN  ANALYSIS 

The  success  rate  pareimeter,  6,  in  advanced  flight  training  is  now  dealt  with  by  a 
standard  Bayesian  analysis  [6].  Two  situations  are  of  interest  in  this  report.  These  two 
situations  involve  the  comparison  of  the  posterior  probability  density  functions  (PPDF) 
for  6  when  the  current  system  selects  students  and  the  PPDF  for  6  when  the  discriminant 
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Table  5.  The  classification  matrix  arising  from  the  logistic  regression  classification  of  451  SNAs.  This  table 
can  be  compared  to  Table  3  which  presents  the  DA  classifications. 


PREDICTED 


Pass 

Fail 

Pass 

344 

88 

ACTUAL 

Fail 

10 

9 

354 

97 

432 

19 

451 


analysis  model  is  used  to  select  students.  The  success  rate  parameter  9  is  considered  to 
be  a  continuous  parameter  with  0  <  ^  <  1.  The  likelihood  of  success  in  advanced  flight 
training  is  assumed  to  follow  a  bino:nial  distribution.  The  merit  of  using  the  DA  model 
discussed  in  this  report  then  hinges  on  the  characteristics  of  its  PPDF  as  compared  to 
the  PPDF  of  the  present  system.  '1  ne  numerical  details  of  this  analysis  are  contained  in 
appendix  C. 

The  prior  probability  density  function  shown  in  Fig.  3  is  based  on  the  calculations 
given  in  Appendix  C  .  This  prior  was  selected  to  represent  our  state  of  knowledge  based 
on  historical  records  for  success  rate  in  advanced  flight  training. 

Of  the  836  SNAs  in  the  data  base,  788  passed  advanced  flight  training,  and  48  failed. 
The  PPDF  constructed  using  this  information  on  success  rate  is  also  shown  in  Fig.  3  as 
the  solid  curve.  The  mean  of  this  PPDF  is  .9434,  the  mode  is  .9443,  and  the  standard 
deviation  is  .0076. 

The  effect  from  the  additional  information  contained  in  the  DA  model  on  the  PPDF 
is  illustrated  in  Fig.  3  as  the  dotted  curve.  Of  the  836  SNAs  in  the  data  base,  451  SNAs 
had  data  on  the  ADHT  test.  Referring  back  to  the  classification  matrix  in  Table  3,  focus 
attention  on  the  column  the  model  predicted  to  pass.  Of  these  348  students,  339  students 
passed  an^  9  failed.  The  mean  of  the  PPDF  using  this  information  is  .9688,  the  mode 
is  ,9709,  iind  the  standard  deviation  is  .0082.  Since  the  DA  model  is  based  on  a  smaller 
sample  size  than  the  present  selection  system,  which  makes  use  of  the  entire  data  base, 
it  is  understandable  that  the  standai'd  deviation  for  the  PPDF  based  on  the  DA  model 
is  correspondingly  larger. 

Therefore,  Fig.  3  shows  the  prior  and  the  two  PPDFs  together  on  the  same  graph 
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Figure  3.  The  prior  and  posterior  probability  density  functions  for  the  success  rate  parameter  9  in  advanced 
fligiit  training.  The  solid  curve  represents  information  using  the  present  selection  system  and  the  dotted 
curve  represents  information  from  the  DA  model. 


Table  6.  Various  Bayesian  confidence  intervals  for  the  posterior  probability  density  functions  for  the  present 
selection  system  and  one  using  the  DA  model. 


Present  System 

DA  Model 

Mean 

94.34% 

96.88% 

90%  Cl 

93.19  to  95.68% 

95.72  to  98.46% 

95%  Cl 

92.90  to  95.86% 

95.40  to  98.78% 

99%  Cl 

92.42  to  96.44% 

94.68  to  99.60% 

for  an  overall  visual  comparison.  The  PPDF  using  information  from  the  ADHT  test  is 
shifted  to  the  right,  illustrating  the  improvement  in  the  success  rate  as  compared  to  the 
current  selection  system. 

Table  6  presents  a  final  characterization  of  the  two  systems  by  calculating  the  90,  96, 
and  99%  Bayesian  confidence  intervals^  for  the  respective  PPDFs.  There  is  no  overlap 
in  the  90%  confidence  intervals  for  the  two  systems.  The  confidence  intervals  for  the 
DA  model  are  wider  than  the  corresponding  confidence  intervals  for  the  present  system 
because  of  the  difference  in  sample  size. 


DISCUSSION 


The  coefficients  of  the  variables  in  the  DA  model  deserve  some  further  comments  in 
light  of  the  cross-tabulation  tables  presented  in  appendix  A.  The  failure  rates  given  there 
are  based  on  the  larger  sample  size  available  for  tlj.e  background  variables  as  opposed  to 
the  smaller  sample  size  when  the  ADHT  was  used  in  the  DA  model. 

The  positive  coefficient  for  the  AQT  aptitude  test  means  that  a  better  score  is  linked 
to  a  lower  probability  of  success  in  advanced  flight  training.  (Remember  that  higher  dis¬ 
criminant  function  scores  result  in  lower  probability  of  success  and  vice  versa).  Although 
it  is  contrary  to  intuition  that  SNAs  with  superior  verbal  and  visuospatial  skills  have  a 
reduced  chance  of  success  in  flight  training,  there  is  some  evidence  in  the  literature  for 
this  phenomenon.  [7] 

*Th«!  terms  credibility  intervid  and  highest  posterior  derksity  (HPD)  arc  also  used  in  the  Bayesian  literature. 
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The  cros&  tabulation  for  AQT  supports  this  finding,  especially  in.  the  range  of  scores 
from  4  to  8  where  the  majority  of  the  data  are  found.  The  probability  of  failure  increases 
from  near  4%  with  an  AQT  score  of  4  to  over  10%  with  an  AQT  score  of  8.  It  re¬ 
mains  somewhat  disconcerting  that  a  test  the  Navy  uses  to  select  individuals  for  training 
eventuates  to  those  having  higher  scores  doing  less  weU  in  advanced  flight  training. 

In  order  of  decreasing  probability  of  success,  the  ranking  of  the  accession  source  vari¬ 
able  (di  and  ^2)  is, 

1.  ROTC/Marines 

2.  Naval  Academy 

3.  AOC/OCS 

The  AOC/OCS  accession  source  has  traditionally  done  less  well  in  all  phases  of  flight 
training,  so  their  ranking  here  is  not  surprising.  Perhaps  slightly  more  surprising,  though, 
is  the  second  place  finish  of  the  Naval  Academy  graduates.  In  our  previous  analysis  of 
success  in  primary  flight  training,  the  Naval  Academy  graduates  had  outperformed  the 
other  two  accesssion  sources. 

The  cross-tabulation  data  show  essentially  no  difference  between  the  Naval  Academy 
graduates  and  the  ROTC/Marines.  The  failure  rate  for  these  two  groups  hovers  around 
4.5%.  The  failure  rate  of  the  AOC/OCS  group  is  clearly  higher  at  around  7%. 

College  major,  (da  and  had  the  ranking  of 

1.  General  Science 

2.  Liberal  Arts/Other 

3.  Engineering/Math 

This  seems  anomalous  as  one  would  expect  the  engineering/math  majors  to  be  the  more 
successful  group.  It  may  be  that  excessively  analytical  tendencies  in  some  few  individu¬ 
als  with  engineering/math  backgrounds  are  antithetical  to  “seat-of-the-pants”  flying  [7], 
although  this  remains  the  merest  sort  of  speculation. 

The  cross- tabulation  data  support  the  superiority  of  the  general  science  majors  vis-a- 
vis  the  other  two  college  major  groupings  with  a  failure  rate  of  about  2.5%  for  general 
science  majors  versus  failures  rates  over  6%  for  the  others.  These  data  do  not,  however, 
support  any  distinction  between  the  engineering/math  majors  and  the  liberal  arts/others 
major  as  the  DA  model  does. 

Gender  and  Age  were  not  included  in  the  DA  model,  but,  interestingly,  note  from 
the  cross-tabulation  tables  that  the  failure  rate  for  women  is  comparable  to  that  for 
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the  men.  Of  course,  the  overall  number  of  women  in  the  data  base  is  extremely  small, 
but  still  accurately  reflects  the  relative  number  of  women  in  the  training  program.  The 
Age  cross-tabulation  table  seems  to  suggest  that  increasing  age  results  in  poorer  training 
performance. 

Returning  to  the  DA  model,  the  dual-task  composite  variable  (ADHTCS)  with  its 
negative  coefficient  was  in  the  expected  direction  with  higher  scores  leading  to  increased 
probability  of  success.  Although  a  strong  relationship  does  not  exist  between  the  ADHT 
composite  score  and  success  in  advanced  flight  training,  the  attrition  rate  can  be  reduced 
if  the  resulting  cost  of  false  rejections  is  acceptable. 

For  example,  the  DA  model  presented  in  this  report  could  achieve  a  50%  reduction  in 
attrition  rate  at  the  cost  of  roughly  a  20%  false  rejection  rate.  That  is,  about  20%  of  the 
SNAs  would  attain  a  discriminant  function  score  that  would  categorize  them  as  failing 
when,  in  fact,  they  would  have  passed.  The  classification  matrix  of  Table  3  illustrates 
this  fact.  Other  desired  attrition  rates  would  entail  different  false  rejection  rates.  These 
inferences  about  a  higher  success  rate  are  related  to  the  particular  variables  chosen  and 
the  specific  linear  composite  of  these  variables  as  formed  by  the  DA  program,  as  well  as 
the  prior  probabilities  and  the  costs  associated  with  correct  and  incorrect  decisions. 

One  rather  naive  approach  to  calculating  cost  savings  is  to  multiply  the  expected 
number  of  SNAs  not  needed  to  be  trained  by  the  cost  per  student  for  the  appropriate 
pipeline.  In  this  numerical  example  we  shall  assume  that  1000  aviators  enter  advanced 
flight  training.  Under  the  present  selection  system  we  would  expect  that  6%,  or  50  SNAs, 
would  at  trite  from  the  program.  If  one  were  to  adopt  the  threshold  cut-off  score  from  the 
DA  model  resulting  in  a  50%  reduction  in  attrition  rate,  only  2.5%,  or  25  SNAs,  would 
at  trite.  Thus,  25  students  would  not  need  to  be  trained,  resulting  in  a  cost  savings.  Table 
7  presents  some  not  unrealistic  training  costs  per  student  for  the  situation  where  the  25 
students  are  evenly  spread  out  over  the  three  pipelines.  Even  considering  the  cost  of  false 
rejection's,  some  significant  monetary  gain  is  achievable. 

Now,  a  straightforward  calculation  similar  to  that  above,  which  might  save  $13. 6M 
per  year  seems  to  support  the  argument  for  implementation  of  the  additional  tests.  The 
actual  situation,  however,  is  probably  more  subtle  than  this  naive  calculation. 

The  following  question  can  be  posed:  “Whai  changes  to  the  training  infrastructure 
would  have  to  be  made  in  order  to  realize  these  projected  savings?”  A  little  thought 
would  seem  to  indicate  that  the  following  factors  would  be  among  the  majo*  components 
involved  in  training  costs. 

1.  The  number  of  aircraft  needed  to  carry  out  the  training  curriculum. 

2.  The  number  of  flight  instructors  to  fly  these  aircraft. 
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Table  7.  One  way  of  calculating  cost  savings  in  advanced  flight  training  when  attrition  is  reduced  from  5% 
to  2.5%.  The  calculation  is  based  on  1000  aviators, 


Pipeline 

Number 

Savings 

Total 

8 

$300,000 

$2.4M 

8 

$500,000 

$4.0M 

JET 

9 

$800,000 

$7.2M 

25 

$13.6M 

3.  The  number  of  hours  flown  during  the  training  curriculum  to  reach  some 

desired  level  of  proficiency. 

4.  The  cost  of  fuel. 

5.  The  cost  of  ground  personnel  and  other  support  personnel. 

6.  The  cost  of  maintaining  the  training  aircraft. 

7.  The  cost  of  simulators  and  simulator  support. 

8.  Billeting  the  students  while  they  are  in  training. 

The  follow-on  question  then  becomes  “How  elastic  are  these  various  factors  in  response 
to  reduced  attritions  in  order  for  savings  to  be  achieved?”  If,  in  fact,  they  are  rather 
inelastic,  then  the  savings  postulated  above  in  Table  7  would  be  difficult  to  realize. 

This  is  simply  an  inquiry  about  the  inherent  rigidity  in  the  training  infrastructure.  If 
the  Navy  has  already  bought  the  planes  for  a  planned  attrition  rate,  set  up  the  billets  for 
flight  instructors  and  ground  personnel,  and  derived  a  curriculum  that  says  students  shall 
fly  these  number  of  hops  and  these  number  of  hours,  then  it  would  be  difficult  to  adjust 
these  factors  because  of  small  changes  in  the  attrition  rate.  Can  the  Navy  actually  buy  2 
fewer  planes,  reduce  5  instructor  billets,  and  fly  30  hours  less  in  the  advanced  curriculum 
simply  because  they  have  25  fewer  pilots  to  train?  We  also  have  to  remember  that  these 
advanced  training  costs  are  spread  out  over  three  different  pipehnes  and  conducted  at 
different  training  bases  where  presumbably  the  costs  axe  again  relatively  fixed. 

Perhaps,  an  easier  way  of  putting  this  into  perspective  is  to  look  at  the  numbers 
involved  in  advanced  flight  training.  If  the  Navy  has  a  training  infrastructure  set  up  to 


17 


train  1000  students,  can  it  really  trim  that  infrastructure  because  it  now  has  “only”  975 
students? 

It  doesn’t  seem  logical  that  the  large  savings  as  indicated  by  the  calculations  made 
above  are  completely  warranted  if  the  rigidity  of  the  training  infrastructure  is  a  reality. 
It  would  be  highly  desirable  to  submit  the  scenario  outlined  above  to  an  expert  in  the 
economic  aspects  of  training  for  a  critical  analysis.  The  decision  to  implement  additional 
selection  tests  clearly  depends  upon  such  an  analysis. 

A  second,  more  subtle,  reservation  about  realizing  savings  due  to  slight  reductions 
in  attrition  rates  concerns  the  “mental  set”  of  the  flight  instructors.  If  some  sort  of 
“corporate  memory”  or  “corporate  culture”  exists  in  the  training  commeinds  that  dictates 
adherence  to  attriting  5%  of  the  students,  then  will  or  c.an  the  instructors  adjust  to 
attriting  only  2.5%  of  the  students?  Is  each  individual  flight  instructor  able  to  fine  tune 
his  assessment  of  flight  proflciency  so  that  a  slightly  smaller  number  of  students  is  attrited 
over  all  pipelines  and  over  all  training  commands?  In  addition,  flight  instructors  might 
not  feel  like  they  are  doing  their  job  if  they  don’t  fail  some  minimum  number  of  students. 
So  the  net  effect  in  the  end  might  be  that,  although  additional  tests  are  effective  and  able 
to  predict  some  of  the  students  who  would  have  failed  in  advanced,  the  flight  instructors 
will  ratchet  up  their  grading  so  that  the  “expected”  number  of  faihires  occur  anyway, 
thus  offsetting  the  effect  of  the  additional  tests. 


CONCLUSION 


A  weak  statistical  relationship  exists  between  a  dual-task  peiforrnance  test,  accession 
source,  college  major,  an  aptitude  test,  and  success  in  advanced  flight  training.  For 
practical  planning  purposes,  it  is  assumed  that  the  decision  to  implement  additional  tests 
would  be  based  on  meaningful  increases  in  the  success  rate.  Information  about  success 
rate  from  our  data  is  neatly  encapsulated  in  the  posterior  probability  density  functions 
shown  in  Fig.  3.  These  PPDFs  show  that  the  success  rate,  using  the  variables  arid  model 
mentioned  above,  is  higher  than  the  success  rate  of  the  present  selection  system,  which 
does  not  take  these  variables  into  account.  More  precisely,  the  90%  Bayesian  confidence 
intervals  for  these  two  density  functions  do  not  overlap,  leading  one  to  the  belief  that 
success  rates  for  these  two  selection  systems  really  are  different.  This  higher  success 
rate  entails  a  cost  in  terms  of  rejecting  some  candidates  who  would  have  been  successful. 
Whether  the  statistical  differences  in  success  rates  can  be  translated  into  actual  dollar 
savings  at  the  training  command  infrastructure  level  remains  an  open  question. 
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APPENDIX  A 


This  appendix  contains  descriptive  data  on  the  backgound  variables  and  training  suc¬ 
cess  in  advanced  flight  training  for  the  836  SNAs  in  our  data  base.  The  tables,  except 
for  the  first,  ai*e  presented  as  cross- tabulation  tables  showing  the  breakdown  of  gender, 
accession  source,  college  major,  AQT,  FAR,  and  age  with  training  success.  The  failure 
percentage  for  each  level  of  these  variables  is  also  presented.  The  total  numbei.,  when 
the  missing  data  are  included,  add  up  to  836  for  each  variable. 


TRAINING  SUCCESS 

Pass 

788 

94.26% 

Fail 

48 

5.74% 

Total 

836 

GENDER 

Level 

Pass 

Fail 

Total 

%  Fail 

Male 

770 

47 

817 

6.75 

Female 

17 

1 

18 

0.56 

Missing 

1 

ACCESSION  SOURCE 

Level 

Pass 

Fail 

Total 

%Fail 

Naval  Academy 

173 

8 

181 

4.42 

ROTC/Marines 

229 

11 

240 

4.58 

AOC/OCS 

381 

29 

410 

7.07 

Missing 

5 

20 


COLLEGE  MAJOR 


Level 

Pass 

Fail 

Total 

%  Fail 

Engineering/Math 

350 

23 

373 

6.17 

General  Science 

113 

3 

116 

2.65 

Liberal  Arts/Other 

320 

22 

342 

6.43 

Missing 

5 

AQT 

Level 

Pass 

Fail 

Total 

%  Fail 

2 

1 

1 

0.00 

3 

18 

4 

22 

22.22 

4 

94 

4 

98 

4.08 

5 

262 

12 

274 

4.38 

6 

206 

11 

217 

5.07 

7 

146 

12 

158 

7.59 

8 

44 

5 

49 

10.20 

9 

17 

0 

17 

0  00 

Missing 

0 

FAR 

Level 

Pass 

Fail 

Total 

%  Fail 

3 

12 

1 

13 

7.69 

4 

23 

3 

26 

11.54 

5 

77 

0 

77 

0.00 

6 

171 

10 

181 

5.52 

7 

144 

6 

150 

4.00 

8 

140 

9 

149 

6.04 

9 

221 

19 

240 

7.92 

Missing 

0 

AGE 


Level 

Pass 

Fail 

Total 

%  Fail 

20 

3 

1 

4 

25.00 

21 

40 

1 

41 

2.44 

22 

303 

17 

320 

5.31 

23 

201 

7 

208 

3.37 

24 

114 

9 

123 

7.32 

25 

61 

7 

68 

10.29 

26 

46 

4 

50 

8.00 

27 

11 

1 

12 

8.33 

28 

4 

1 

5 

20.00 

29 

2 

0 

2 

0.00 

30 

1 

0 

1 

0.00 

Missing 

2 

APPENDIX  B 


The  SPSS/PC+  Advanced  Statistics  Manual  explains  in  general  terms  ho\v  cases  are 
classified  by  the  discriminant  analysis  program.  “A  case  is  classified,  based  on  its  dis¬ 
criminant  score,  in  the  group  for  which  the  posterior  probability  is  largest.”  [5].  Bayes’s 
Theorem  is  used  to  calculate  the  posterior  probability 

^  P(O|G.0  .  P(C,) 

‘  tL  P(DIGd  .  P(Gi 

where  P(6?,|D)  is  the  posterior  probability  for  Group  i,  P(D|G,)  is  the  likelihood  of  the 
data  given  Group  i,  and  P(G,)  is  the  prior  probability  of  Group  i. 


For  the  two-group  case,  which  is  the  focus  of  our  attention  in  this  study,  this  formula 
explicitly  reduces  to 


P(GilD) 

P(C?2|P) 


P(DlGi)  *  P(G^) 


P(BlGt)  *  P(Gi)  +  P(D|(72)  *  P(G2) 

l-P(GrlD) 


(1) 

(2) 


SPSS  apparently  uses  the  minimum  nile  [8],  which  is  the  same  as  using  the  maxi¬ 
mum  posterior  probability. 


P(DIG,)  =  exp-^ 

Since  x’  =  (~V”) 
and  <7  =  1 

P(D|Gi)  exp-^^^i£ 

F(VIG2)  = 


The  discriminant  scores,  i,-,  are  calculated  by  the  SPSS/PC-I-  program  so  that  a  = 
1,  Xi  is  the  mean  of  the  students  who  passed,  and  X2  is  the  mean  of  the  students  who 
failed  on  the  canonical  discriminant  function.  Substituting  these  values  for  P(Z>|Gi)  and 
PlDjG^)  and  where  Gi  =  PASS  and  G2  =  FAIL  in  equation  1  yields. 


P(PA551Z))  = 


_ exp-^^*  PjPASS) _ 

exp  *  P{PASS)  -f  exp  ♦  P{FAIL) 
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A  numerical  example  using  this  formula  for  the  calculation  of  the  posterior  probability 
of  belonging  to  the  PASS  or  FAIL  group  follows: 


The  discriminant  score  for  the  hypothetical  student 

X  =  .1638 

Group  mean  of  discriminant  scores  for  PASS 

Si  =  -.03637 

Group  mean  of  discriminant  scores  for  FAIL 

X2  =  .82694 

Prior  probability  of  PASS 

P{PASS)  =  .58 

Prior  probability  of  FAIL 

P{FAIL)  =  .42 

P(PA5S!a:.- _  .1638)  -  ^  gg  ^  _  L^e3s^2e?^^_  ^ 

.9802  *  .58 

(.9802  *  .58)  +  (.8026  *  .42) 

=  .627C 

Therefore,  the  posterior  probability  of  a  PASS  in  advanced  flight  training  given  that 
a  student  achieved  a  discriminant  function  score  of  .1638  =  .6278.  FVom  equation  2 
the  posterior  probability  of  a  FAIL  =  .3722.  The  probability  of  belonging  to  the  PASS 
group,  given  a  discriminant  score  of  .1638,  is  about  63%  and,  therefore,  this  student 
will  be  classifled  by  the  program  as  a  PASS  because  the  PASS  group  has  the  maximum 
posterior  probability. 

The  prior  probabilities  parameter  for  the  PASS  and  FAIL  groups  was  changed  in  the 
DA  program  through  trial-and-error  until  the  classification  matrix  yielded  as  close  to 
a  50%  reduction  in  failures  as  possible.  The  values  arrived  at  were  P(PASS)=.58  and 
P(FAIL)=.42,  as  given  above.  This  parameter  takes  into  account  not  only  the  prior 
probabilities,  but  also  the  payoff  matrix  for  all  four  possible  decisions.  It  reflects  the 
willingness  to  tradeoff  false  rejections  for  a  reduced  number  of  failures  and  explains  why 
the  prior  probabilities  were  not,  for  example,  simply  P(PASS)=.95  and  P(FAIL)=.05. 

The  threshold  discrinunant  function  score,  (xthreshoid),  above  which  a  student  v/ill  be 
classified  as  a  FAIL,  and  below  which  the  student  will  be  classified  as  a  PASS  can  be 
determined 


P(PASS\xtkre,hoid  =7)  =  -50 
P{PASS\xthre»hotd  =  -769)  =  .50 

Any  SNA  who  gets  a  discriminant  function  score  above  .769  will  be  predicted  to  fail 
advanced  flight  training;  any  discriminant  fimction  score  below  .769  will  be  predicted  to 
pass. 

In  addition  to  this  direct  approach,  logistic  discrimination  using  the  discriminant  func¬ 
tion  scores  can  be  used  as  well  to  calculate  the  same  posterior  probabilities.  It  is  an 
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interesting  exercise  to  malce  the  comparison  between  the  parameters  used  above  and  the 
logistic  discrimination  parameters.  This  derivation  is  shown  in  the  final  section  of  the 
appendix. 

Ultimately,  the  goal  is  to  derive  an  expression  for  the  posterior  probability  for  the  two 
groups  as  a  logistic  function. 


P(Gt\D)  = 
PiG2\D)  = 


gO;  +  pz 

1  +  gO  +  /?a: 
1 

1  +  gO  + 


By  using  discriminant  function  scores  to  classify  SNAs,  it  can  be  safely  assumed  that 
these  scores  are  normally  distributed  with  a  standard  deviation  equal  to  one. 

To  start  the  derivation  of  the  logistic  function  for  the  posterior  probabilities  refer 
back  to  equation  1.  Dividing  both  the  numerator  and  the  denominator  of  equation  1  by 
P(D|G2)  +  P((?2)yields, 

p(D\Gi)*p(ai) 


P{Gi\D)  -  ,  , 

Within  this  expression,  focus  attention  on  the  likelihood  ratio, 


P(P|gi,^ 

P{D\G2) 


P{D\G,)  = 
P(D|<?2)  = 

Since  (T\  — 

P(D\Gx) 

P{D\G2) 


criv27r 

a2v27r 
<T2  =  1 

g-i(®-/l3)* 


=  e  2 


Expeinding  both  quadratic  terms  in  the  exponent: 

(x-pLi)^  =  -2xni+ 

(X  -  112^  =  x*  -  2x/i2  +  ^p2 
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2  2 


Let  ao 

/5 

P{G,\D) 


=  X/ii  -  XH2  -  ^(/ij  -  ^ll) 

—  (/^l  “  M2)(/^1  +  ^^2) 

=  -  /^2)  -  2(/^l  - /^2)(/^l  + /i2) 


=  e 


/^1  -  ^^■2 
ao+/)x 


.0(o+/3x  *  ^(Qi) 

^  *  P{Gi) 

goo+fix  ^  n2ii  4.  i 
^  P(G2)  ^  ^ 


The  posterior  probability  is  now  very  close  to  the  required  form.  The  final  step  is  to 
transform  the  expression 

e^o  +  /3x  ^P(^Gi)/P{G2) 

The  general  approach  is  to  take  inverse  operations  in  succession  and  form  a  new  constant. 
Taking  the  natural  log  (In)  and  exponentiating  are  two  such  inverse  operations.  The 
expression 

e°‘°  +  l^^‘*P{Gi)/P(G2) 

is  rewritten  as 

e®  ♦  y\/y2 


Step 

Operation 

Result 

■  1 

1 

Take  natural  log 

Me®  *  yi/y2)  =  X  +  Myi/j/2) 

2 

Form  exponential 

gX  +  MyiM)  =  e^o  +  (3x  +  ln[(P(Gi)/P(02)] 

3 

Form  new  constant 

ai=ao  +  ln[P(G0/P(G2)] 

4 

Posterior  Probability 

„ai  +  Px 

It  is  quite  easy  at  this  point  to  show  that 

P(G2\D)  =  l-P(G.P) 
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1  ^  g£»l+/?®  gOl+Z?* 

~  1  +  1  + 

1 

“  l^.eai+/3i 

The  following  numerical  example  shows  that  this  derivation  provides  the  same  poste¬ 
rior  probability  for  a  given  discriminant  score,  P(P^S5|a;  =  .1638),  as  calculated  before: 

Oi  =  ao  +  HP(PASS)/PiFAIL)] 

Oo  =  —-^if^PASS  —  f^FAIL){t^PASS  +  f^FAIL) 

=  -^(-.03637  -  .82694)(-.03637  +  .82694) 

=  .3413 

oi  =  .3413 -1- ln[.58/.42] 

=  .6640 

P  =  fiPASS  “  f^FAIL 

=  -.03637  -  .82694 
=  -.8633 

ai+/3x  =  .6640- .8633(.  1638) 

=  .5226 

pon+pa 

PiPASSU  ^  .ms)  = 

g.6226 

”  1  +“c-®226 

=  .6278 
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APPENDIX  C 


The  calculations  in  this  appendix  show  how  the  prior  and  posterior  probability  den¬ 
sity  functions  presented  in  Fig.  3  were  derived.  The  numerical  details  of  the  Bayesian 
confidence  interval  calculations  are  also  presented. 

The  derivation  of  restdts  presented  in  the  main  section  of  the  report  follows  a  standard 
Bayesian  treatment  of  a  binomial  distribution  with  a  continuous  parameter  [6].  A  beta 
distribution  was  chosen  as  a  natural  conjugate  prior  for  6^  the  success  rate  parameter.  A 
beta  distribution  is  characterized  by  two  parameters,  a  and  /?,  both  gi-eater  than  zero. 
The  functional  form  for  the  prior  distribution  is 

The  prior  distribution,  g(6),  chosen  for  this  analysis  had  a  =  95  and  /3  =  5.  The 
expectation  of  6  is 


The  variance  of  6  is  defined  as 


var(e)  -  ■  Vfl  .  .N  =  *0005 

(a-f  1) 

The  standard  deviation  of  6  is,  therefore,  .0217.  These  values  of  a  and  /?  were  selected 
to  represent  the  state  of  knowledge  for  success  rate  in  advanced  flight  training  based  on 
historical  records. 

The  beta  function,  B(a,/?),  is  defined  as 

The  ordinate  of  g(9)  can,  therefore,  be  calculated  by 

g{e)  = 

_  (a  +  l3-l)\ 

*  (a -1)!  (/?-!)! 


k2  —  a  —  1 

^3  =  /?  -  1 
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A  numerical  example  of  calculating  the  ordinate  for  ihe  prior  density  for  a  specific 
value  of  0,  for  example,  ^  =  .92,  would  be 


g(&  =  .92)  =  ki  .92^  {1  -  .92) 


,  99! 

“  9414! 

=  357,615,720 

=  357, 615, 720-^.92®^*. OS'* 

=  5.7787 


The  ordinate  for  6  =  .92  shown  in  Fig.  3  is  5.78. 

The  posterior  density  function,  that  is,  the  density  function  for  0  after  considering  ad¬ 
ditional  data,  because  of  the  convenient  mathematical  properties  of  the  natural  conjugate 
prior,  is 

_ ]; _ _  ^yi-y+/3-l 

B(y  -i-  Q!,n  -  g  +  jS) 

This  is  seen  to  be  of  the  same  fimctional  form  as  for  the  prior  density  function,  but 
with  the  number  of  successes  and  the  nmnber  of  failures  in  the  new  data  included.  The 
number  of  successes  in  the  new  information  is  p,  and  the  number  of  failures  in  the  new 
information  is  n  —  y. 


For  a  numerical  example,  consider  the  construction  of  the  posterior  probability  density 
function,  /i(^|y),  for  the  present  selection  system.  The  ordinate  of  the  posterior  density 
function  at  0  =  .955  is. 


Total  sample  size 
Number  of  PASSes 
Niunber  of  FAILures 
Prior  density 
Prior  density 
Posterior  density 
Posterior  density 


n  =  836 
y==788 
n  --  y  =  48 
a  ~  95 
/?  =  5 

y  -f  a  —  1  =  882 
n-y-|-/?-l  =52 


/i((?  =  .955|y  =  788)  = 


1 


-  6) 


52 


5(883,53) 

=  7.8339  *  10®®  *  .955®®^  *  .045®^ 

:=  16.7495 

In  the  second  situation,  the  PPDF  was  constructed  for  6  when  we  used  information 
from  the  ADHT  dual  task  and  certain  biographical  data.  To  calculate  the  ordinate  for 
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this  PPDF  at  the  same  theoretical  pass  rate,  h{9  =  .955|y  =  339), 


Total  sample  size 

n  =  348 

Number  of  PASSes 

y  =  339 

Niimber  of  FAILures 

n  —  y  =  9 

Prior  density 

a  =  95 

Prior  density 

0  =  5 

Posterior  density 

y  +  a  —  1  =  433 

Posterior  density 

n-y-t-^-l  =  13 

h{e  =  .955|j/  =  339) 


1 

B(434, 14) 

I. 6623  ♦  10 

II. 3222 


^“^^(1  -  0) 


13 

♦  .045^^ 


In  this  final  section  of  appendix  C  we  show  how  the  confidence  intervals  were  cal¬ 
culated.  One  of  the  advantages  of  the  Bayesian  approach  is  the  direct  meaning  of  the 
concept  of  a  confidence  interval.  It  is  simply  the  desired  area  of  the  posterior  probability 
density  function,  usually  calculated  in  equal  increments  starting  from  the  mode  of  the 
PPDF.  The  area  of  the  PPDF  for  a  95%  Bayesian  confidence  interval  is  represented  by 


S(j/ -f  a,  n  -  y -f- /?) 


^V+a-1^2  _  ^)n-V+/3-l 


de  -  .95 


where  I  —  lower  limit  of  6  and  u  =  upper  limit  of  9  such  that  the  value  of  the  integral 
equals  the  required  confidence  interval,  namely  .95.  The  Mathcad  software  package  [9] 
W2is  used  to  vary  the  lower  and  upper  limits  until  the  desired  value  of  the  integral  was 
reached.  The  lower  and  upper  limits  were  systematically  moved  in  equal  increments  from 
the  mode  of  the  particular  PPDF. 
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